🕷️️ Job Radar • SCRAPING

freelancer.com 2026-04-22 🟡

🔹 Automate product data extraction from e-commerce websites into structured formats.
👤 Client: 🇮🇳 Coimbatore, India Member since 2024-11-14
💰 Price: $247 Average bid
🚩 Problem: Need a Python script that efficiently scrapes product details while respecting web scraping best practices and handling dynamic content.
📦 Existing: Not specified

Specifications:

[Target] E-commerce website(s) with varying HTML structures and JavaScript rendering needs.
[Method] Modern Python (3.10+), using libraries like requests, BeautifulSoup, Scrapy, Selenium or Playwright for dynamic content handling.
[UI/UX] Configurable via a single JSON config file for site URLs, selectors, and output paths.
[Stack] Python 3.10+, requests, BeautifulSoup, Scrapy (optional), Selenium (optional), Playwright (optional), pandas, logging.
[Security] Headless operation, user agent rotation, delay mechanisms to avoid blocking.
[Format] CSV and JSON for structured data export.

Workflow:

Analyze the sample e-commerce site(s) provided by the client to determine the best scraping approach (requests + BeautifulSoup vs. Selenium/Playwright).
Set up a Python virtual environment, install required libraries, and create a basic project structure.
Write initial code for data extraction using requests or Scrapy for static content, with fallback to Selenium/Playwright if dynamic content is detected.
Implement logging mechanisms to track each run’s progress and errors.
Create a configuration file template that allows users to specify site URLs, HTML selectors, and output paths.
Test the script on the provided sample sites to ensure it runs without errors and exports data in CSV and JSON formats.
Document the code with clear comments for easy extension to additional sites.
Write a README detailing setup instructions, required packages, and how to add new selectors.